Skip to content

Drop Marshal from persistent storage in favour of Deriving_Json#859

Draft
balat wants to merge 88 commits into
masterfrom
deriving
Draft

Drop Marshal from persistent storage in favour of Deriving_Json#859
balat wants to merge 88 commits into
masterfrom
deriving

Conversation

@balat

@balat balat commented May 29, 2026

Copy link
Copy Markdown
Member

Summary

Replace Stdlib.Marshal with Deriving_Json for all persistent
storage in Eliom (Ocsipersist-backed cookies, session groups,
session data and Reference.eref). Adds a [@@deriving json]
annotation on the types involved and bumps the on-disk table names
where the format changes. Intended for the Eliom 13 release.

This is a breaking change. User code must now pass a JSON codec to
persistent references:

Reference.eref ~persistent:("name", [%json: t]) v
State.create_persistent_table ~scope ~json:[%json: t] "name"

Commits

  • Serialise persistent session groups with Deriving_Json — replaces
    Marshal.{to,from}_string for perssessgrp with a nominal payload
    record and adds [@@deriving json] on cookie_level.
  • Switch persistent-cookie table to Ocsipersist.Column.Json — bumps
    the cookie table suffix from _v5 to _v6 and annotates the cookie
    type + all its dependencies (scope_hierarchy, user_scope,
    full_state_name, timeout, perssessgrp) with [@@deriving json].
  • Switch session-group table to a typed Column.Json table — replaces
    Ocsipersist.Polymorphic with a typed functorial table; renames the
    table to __eliom_session_group_table_v2; annotates nbmax.
  • Drop Marshal from Reference and State persistent tables — the
    breaking API change. Reference.eref and State.create_persistent_table
    now take a Deriving_Json.t codec. The vestigial int64 column on
    every session value is dropped. Common.Ocsipersist.Polymorphic and
    Persistent_tables.create are removed.

Downstream

Companion PRs:

  • ocsigen-start: deriving branch (codec for Stringset in tips,
    Os.Date.local_calendar_json, demo template update, opam constraint
    relaxed, stale pin-depends removed)
  • ocsigen-toolkit: deriving branch (only opam constraint relaxed)
  • os_template: deriving branch (demo_ref.eliom codec update)

Test plan

  • dune build @check passes on the deriving branch
  • dune build passes on ocsigen-start and ocsigen-toolkit pinned
    against this branch
  • Manual smoke test of a persistent session round-trip in a real
    app (e.g. ocsigen-start)
  • Confirm that the Eliom manual + tutorial sections about
    Reference.eref have been updated (separate PRs)

balat added 25 commits June 22, 2026 14:37
Pure file renames, no content changes. For example:
  eliom_service.server.ml → service.server.ml
  eliom_content.eliom → content.eliom

Files NOT renamed (name conflicts with Stdlib/external libs):
  eliom_form (conflicts with Js_of_ocaml.Form)
  eliom_lazy (conflicts with Stdlib.Lazy)
  eliom_react (conflicts with react library)
  eliom_uri (conflicts with uri library)

Internal modules also kept unchanged:
  eliom_common, eliom_types, eliom_runtime, *_base, *_core, etc.

This commit does not compile: reference updates follow.
Update all internal references from Eliom_xxx to the new short names.
For example:
  Eliom_service.create → Service.create
  Eliom_parameter.int → Parameter.int

References updated in:
  - Renamed source files (74 files)
  - Internal modules (eliom_common, eliommod_*, etc.)
  - PPX code (ppx_eliom_server.ml, ppx_eliom_utils.ml)

References NOT changed (modules not renamed):
  Eliom_form, Eliom_lazy, Eliom_react, Eliom_uri
  Eliom_common, Eliom_runtime, Eliom_types, and all *_base/*_core modules
Create/update the Eliom module as the primary entry point:
  Eliom.Service, Eliom.Registration, Eliom.Content, etc.

Modules with name conflicts use the old names:
  Eliom.Form = Eliom_form  (Js_of_ocaml.Form conflict)
  Eliom.Lazy = Eliom_lazy  (Stdlib.Lazy conflict)
  Eliom.React = Eliom_react (react library conflict)
  Eliom.Uri = Eliom_uri    (uri library conflict)

Create client-side Eliom module (previously server-only).
Update dune: modules_without_implementation for renamed sig modules.
Regenerate dune.server and dune.client.
Provide backward-compatible module names (Eliom_service, etc.) in
separate libraries for existing code that hasn't migrated to the
new Eliom.Service style.

Usage: add eliom.compat.server (or .client) as a library dependency.

These libraries are intended to be deprecated and eventually removed
once downstream code has migrated.

Server: 22 compat modules (16 .ml + 6 sig-only .mli)
Client: 19 compat modules (12 .ml + 7 sig-only .mli)
- Migrate Eliom_X module references to Eliom.X (Eliom_service.create
  → Eliom.Service.create, Eliom_registration.App → Eliom.Registration.App,
  Eliom_content.Html.F → Eliom.Content.Html.F, etc.)
- Rename PROJECT_NAME_main.eliom → PROJECT_NAME_server.eliom
  (executables: testapp_server.bc, testapp_server.exe)
- Update Makefile.app and dune targets accordingly
- Eliom_xxx → Eliom.Xxx in source files
- api.odocl: use new short module names
- indexdoc: update documented module list
- filelist.ml: update module list for packaging
- _tags: disable warning 34 (OCaml 5.4), add logs.browser package
Define a named module type Content.Html.T (built from Tyxml's
Html_sigs.Make(Xml)(Svg.F.Raw').T plus Eliom's LINKS_AND_FORMS) and
use it as the parameter signature of Tools.Make instead of
[module type of Content.Html.F].

Avoids the OCaml strengthening bug that triggers when [module type of]
captures wrapped module paths through an alias.
Use dune's [(wrapped (module_name Eliom))] feature to expose the
public modules as [Eliom.Service], [Eliom.Content], [Eliom.Client],
... instead of the historical flat [Eliom_xxx] names.  Update PPX
client and dune configuration accordingly.
The module was incorrectly left with its old name despite being
visible to users via PPX-generated type annotations.

Also rename the local Client_value module in eliom_client_core.client.ml
to Client_value_registry to avoid shadowing.
Accessible as Eliom.Syntax via wrapping. Used by the PPX in
generated code (Eliom.Syntax.client_value, Eliom.Syntax.to_poly, etc.)

Also adds Syntax.to_poly (previously only in Lib.to_poly) since the
PPX generates Eliom.Syntax.to_poly in user code.
ppx_eliom_client.ml generates code referencing Eliom_client_core.
With wrapping, this must be Eliom.Eliom_client_core for user code.
Use eliom_expr to conditionally prefix with Eliom. when not -internal.
Visible to users via PPX-generated code (register_client_closure,
open_client_section, close_server_section, get_escaped_value).
…odules

The injection expression was hardcoded as Client_core.Syntax_helpers.get_injection
instead of going through eliom_expr which prepends Eliom. when not in -internal mode.
Breaking change: Eliom.run is now Eliom.App.run due to module wrapping.
The auto-generated wrapper module cannot contain val declarations,
so this cannot be covered by eliom-compat.
Define eliom-compat as its own (package ...) in dune-project (was
previously an [eliom.compat] sub-library) so that odoc no longer
collides between the modernized [Eliom.Xxx] names exported by [eliom]
and the legacy [Eliom_xxx] aliases exported by the compat lib.

The two libraries are renamed from eliom.compat.{server,client} to
eliom-compat.{server,client}.

Existing dune executables/installs that were implicit when there was
only one package now need an explicit (package eliom) stanza.
The old ocamlbuild-based doc system could not handle wrapped modules.
New build/gen_wikidoc.sh script:
- Uses dune-compiled .cmi files from _build/
- Creates short-name .cmi aliases (Bus.cmi -> include Eliom__Bus)
  so ocamldoc can resolve cross-references
- Gets include paths from dune's merlin config
- Generates wiki docs for both server and client
- Remove src/ocamlbuild/ (eliom.ocamlbuild plugin)
- Remove pkg/ (topkg packaging system)
- Remove build/build.ml, build/doc.ml, build/_tags (ocamlbuild config)
- Remove _tags, src/_tags, src/ppx/_tags, README.build
- Remove js_of_ocaml-ocamlbuild dependency from dune-project/opam
- Clean up ocamlbuild references in comments

The doc generation now uses build/gen_wikidoc.sh (previous commit).
The distillery templates in pkg/distillery/ are preserved.
When -internal-prefix Os is passed, the client PPX strips the Os.
wrapper prefix from type paths read from server .cmo files. This is
needed for compiling wrapped libraries like ocsigen-start that depend
on Eliom.
The eliom.opam.template is consumed by dune to append the
x-ci-accept-failures stanza into the generated eliom.opam.
Accessible as Eliom.Common from user code.
balat and others added 6 commits June 25, 2026 19:34
Keep the (hosted ...) table in sync with master: add the js_of_ocaml/tyxml
(subdir layout) and reactiveData (root) entries so cross-project links point
to ocsigen.org. Avoids reverting them when this doc branch merges.

Requires wodoc with the `subdir` layout (ocsigen/wodoc#4).
Keep the (hosted ...) table in sync with master: add (lwt lwt subdir) now
that Lwt's docs are deployed under the standard per-package layout, so this
modernization branch doesn't drop the Lwt redirect when it reaches master.
Eliom had two overlapping landing pages: overview.mld (the ocsigen.org
landing) and index.mld (the ocaml.org landing), both an "Eliom" introduction
with installation and getting-started examples. Keep the richer overview as
the single home page by renaming it to index.mld, and drop the redundant one.

On ocaml.org the documentation landing (/p/eliom/latest/doc/) is rendered
from index.mld; add @children_order so the manual pages appear in the sidebar
in the reading order of the reference manual instead of alphabetically. Point
the ocsigen.org landing at index.html.
The package home page (index.mld) is the ocsigen.org landing but was not
reachable from the left navigation. Add an "Overview" entry pointing at it,
before the Introduction.
Bring the deriving branch in line with master: convert hardcoded
ocsigen.org/<project> links (here version-less and root-relative) to odoc
cross-package references, quoting hyphenated page names, and repair the two
broken empty-page links (/tyxml/.html, /lwt/.html) to the project home. Keeps a
future merge from reintroducing the old URLs.
The API docs are now built with odoc/wodoc (see doc/wodoc and the
Documentation CI workflow); the gen_wikidoc.sh ocamldoc->wiki script,
the make wikidoc/doccp targets and the README section are no longer
used. The indexdoc menus are kept (still consumed by doc/wodoc).
@balat
balat changed the base branch from modernize to master June 25, 2026 18:24
Eliom on this branch targets the Ocsigen Server 8.x API; the dependency
bound still allowed only 7.x (< 8.0), which made eliom uninstallable
against ocsigenserver 8.0. Bump the bound to (>= 8.0) (< 9.0).
balat and others added 18 commits June 26, 2026 16:44
The repository is already formatted with ocamlformat 0.29.0 in practice
(editor format-on-save), but `.ocamlformat` still pinned 0.28.1, so the
`lint-fmt` CI job — which installs the pinned version — failed on the
0.29.0-formatted code.

Bump the pinned version to 0.29.0 and reformat the few remaining files
that still carried 0.28.1 formatting (via `dune fmt`).
Ocsigen Server 8 removed the deprecated helpers id, leftright, map_filter
and the Option module from its baselib. Replace them with their stdlib
equivalents instead of relying on shims:

  - id            -> Fun.id
  - List.map_filter -> List.filter_map
  - Lib.Option    -> stdlib Option (map/iter, identical signatures)
  - leftright     -> Either.t, with Left/Right -> Either.Left/Either.Right

Also drop the now-unused `open Lib` in mod_sessiongroups.mli.
Eliom now targets the Ocsigen Server 8.x API. Bump the bound in
dune-project from (>= 7.0) (< 8.0) to (>= 8.0) (< 9.0) and regenerate
eliom.opam.
Released ocsipersist still targets the flat ocsigenserver module names; the
master pin tracks the Ocsigen Server 8 API.
CI: pin ocsipersist#master for Server 8
Require ocsigenserver >= 8.0 (and < 9.0)
# Conflicts:
#	.github/workflows/doc.yml
#	CHANGES
#	doc/client.indexdoc
#	doc/server-outputs.mld
#	dune-project
#	eliom.opam
#	src/lib/client.client.ml
#	src/lib/form_sigs.shared.mli
Eliom now targets the js_of_ocaml 6.4 API (merged from master). Bump the
js_of_ocaml/wasm_of_ocaml lower bounds to 6.4.0.
The Eliom PPX wrapped every processed module in set_global true/false,
including plain .ml/.mli files when a project applies the PPX
library-wide (as opposed to Eliom's own build, which feeds only
.eliom/.eliomi to the PPX). In that global context a server-only .ml
that creates services at toplevel makes Eliom_service.create allocate
a client value (no_client_fun's `ref None`) which, since the module is
never linked on the client, is reported at runtime as "Code generating
the following client values is not linked on the client".

Make the PPX a no-op on .ml/.mli: only .eliom/.eliomi are processed as
client-server code, matching the intended meaning of the extensions.
# Conflicts:
#	.github/workflows/doc.yml
A persistent value or client-sent datum that fails Deriving_Json
deserialisation -- e.g. data written by a pre-Eliom-13 (Marshal-based)
release, or a stale/corrupt cookie -- must not raise an uncaught
exception: it otherwise 500s every request until the user clears their
cookies.

Catch the deserialisation Failure and fall back to a safe default:
- getperssessgrp: treat as an expired session (Eliom_Session_expired);
- request-info parsing (tab cookies, client process info,
  expecting-process-page, cookie substitutes): treat the datum as absent;
- Reference.get for persistent references (Per / Ocsiper / Ocsiper_sit):
  reset the reference to its default value.
Persistent references (Site scope) and persistent state tables
(group/session/process) are created through
Persistent_tables.create_json under their bare logical name, which is
the same name the pre-13 Marshal-encoded tables used. On upgrade the
JSON code would otherwise read the stale Marshal rows (and, on
PostgreSQL, try to write JSON text into the old bytea column).

Give these JSON tables a "_json_" prefix so they never collide with
the old Marshal tables: the stale tables are left orphaned and Eliom 13
starts from fresh, correctly typed tables. "_json_" begins with '_', a
valid unquoted PostgreSQL identifier, so no identifier quoting is
needed in Ocsipersist.

The global reference store and the cookie/session-group tables are
already orphaned (distinct store prefix / version suffix), so only
create_json needs the prefix.
site_init runs the register_site_init callbacks under set_global true
(creating client values such as set_ignored_params and the Comet fallback
services) but never closed a server section. Those buffered values were
therefore captured by the next loaded module's close_server_section. When
that module was not linked on the client (e.g. an unused Ot widget whose
server side is still loaded), its section was never consumed, so the
values leaked to the runtime check ("Code generating the following client
values is not linked on the client") and, more importantly,
set_ignored_params was never actually applied on the client.

Close a dedicated section (__eliom_site_init__) right after the callbacks
so the values are attributed there, and consume it on the client in
do_request_data (before the check), which initialises them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant